{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Semantic Chunking\n",
    "> Splits the text based on semantic similarity.<br>\n",
    "根据语义相似性拆分文本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "meow meow🐱 \n",
      " meow meow🐱 \n",
      " meow😻😻\n",
      "\n",
      " sss\n",
      " aaa\n"
     ]
    }
   ],
   "source": [
    "from langchain_experimental.text_splitter import SemanticChunker\n",
    "from langchain_openai.embeddings import OpenAIEmbeddings\n",
    "\n",
    "# This is a long document we can split up.\n",
    "with open(\"../data/meow.txt\") as f:\n",
    "    state_of_the_union = f.read()\n",
    "\n",
    "text_splitter = SemanticChunker(OpenAIEmbeddings())\n",
    "docs = text_splitter.create_documents([state_of_the_union])\n",
    "print(docs[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Percentile\n",
    "The default way to split is based on percentile. In this method, all differences between sentences are calculated, and then any difference greater than the X percentile is split.\n",
    "\n",
    "默认的拆分方式是基于百分位数。在这种方法中，计算句子之间的所有差异，然后将大于 X 百分位数的任何差异分开。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "meow meow🐱 \n",
      " meow meow🐱 \n",
      " meow😻😻\n",
      "\n",
      " sss\n",
      " aaa\n"
     ]
    }
   ],
   "source": [
    "text_splitter = SemanticChunker(\n",
    "    OpenAIEmbeddings(), breakpoint_threshold_type=\"percentile\",breakpoint_threshold_amount=0.1\n",
    ")\n",
    "docs = text_splitter.create_documents([state_of_the_union])\n",
    "print(docs[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standard Deviation \n",
    "标准差\n",
    "\n",
    "In this method, any difference greater than X standard deviations is split.\n",
    "\n",
    "在此方法中，任何大于 X 个标准差的差值都将被分割。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "meow meow🐱 \n",
      " meow meow🐱 \n",
      " meow😻😻\n",
      "\n",
      " sss\n",
      " aaa\n"
     ]
    }
   ],
   "source": [
    "text_splitter = SemanticChunker(\n",
    "    OpenAIEmbeddings(), breakpoint_threshold_type=\"standard_deviation\"\n",
    ")\n",
    "docs = text_splitter.create_documents([state_of_the_union])\n",
    "print(docs[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interquartile \n",
    "四分位数\n",
    "\n",
    "In this method, the interquartile distance is used to split chunks.\n",
    "\n",
    "在这种方法中，四分位距用于分割块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "meow meow🐱 \n",
      " meow meow🐱 \n",
      " meow😻😻\n",
      "\n",
      " sss\n",
      " aaa\n"
     ]
    }
   ],
   "source": [
    "text_splitter = SemanticChunker(\n",
    "    OpenAIEmbeddings(), breakpoint_threshold_type=\"interquartile\"\n",
    ")\n",
    "docs = text_splitter.create_documents([state_of_the_union])\n",
    "print(docs[0].page_content)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain0_1",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
